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Abstract 

The introduction of engineering practices within the Next Generation 
Science Standards provides technology educators with opportunities to help 
STEM educators infuse engineering design within a core curriculum. The 
introduction of teaching engineering design in early elementary grades also 
provides opportunities to conduct research investigating how young students use 
engineering design as a way to solve problems. There is a need for research to 
assess how students experience engineering design as a pedagogical approach to 
learning science. This article will feature research on elementary students’ 
cognitive strategies used during engineering-design science activities. We 
adopted the concurrent think-aloud (CTA) protocol analysis method to capture 
how students conceptualize design and enhance science learning. During the 
2012-2013 school year, we video recorded 66 CTA sessions, and this study 
examines six of those sessions. NVivo (Version 10) was used to code each video 
using common cognitive strategies categorized by Halfin (1973). Research 
findings indicate that participants increased the amount of time spent on 
mathematical thinking by 34% when given a math-specific design task. Pre- and 
post-tests showed that participants gained significant science content 
knowledge. However, we also confirmed that participants struggled with 
applying accurate mathematical and scientific knowledge to solving the given 
design problem. 

Keywords: concurrent think-aloud protocol; design cognition; transfer of 
learning 

Design is a core component of technology education (Lewis, 2005). 
Engineers, designers, and others in technology design and create solutions to 
given problems. Therefore, technology educators have been implementing the 
engineering-design approach as an effective way to teach technology. Although 
technology education is putting greater emphasis on engineering design (Hill, 
2006; Lewis, 2005; Wicklein, 2006), recently, K-12 science education in the 
United States has proposed the teaching of engineering practices alongside the 
teaching of science practices. For example, the Framework for K-12 Science 
Education (National Research Council [NRC], 2012) includes engineering- 
design learning standards. The framework provides a strong platform for 
teaching engineering and technology contexts to enhance students’ science 
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learning. The document states: 

Engineering and technology provide a context in which students can test 
their own developing scientific knowledge and apply it to practical 
problems; doing so enhances their understanding of science—and, for 
many, their interest in science—as they recognize the interplay among 
science, engineering, and technology. We are convinced that engagement in 
the practices of engineering design is as much a part of learning science as 
engagement in the practices of science [(National Academy of Engineering 
and National Research Council, 2009)]. (p. 12) 

Many states, including Massachusetts and Minnesota, have created 
academic standards requiring students to engage in the engineering-design 
process and to explore the nature of technology and engineering practices within 
science standards (Robelen, 2013). Conceptually, the driving force behind these 
educational reforms is the emphasis on students developing the abilities to 
define problems by asking questions, create and apply models, generate plans, 
engage in design challenges, and apply evidence-based scientific knowledge to 
create and select the best possible solution to a problem (NRC, 2012). 

With the introduction of the Next Generation Science Standards to the 
elementary science classroom, technology educators can use their long history 
of design study in the secondary grade level to investigate the use of engineering 
design with elementary students. This will provide technology educators with a 
better understanding of how young students solve problems using the 
engineering-design approach. In addition, technology educators have shown that 
engineering design not only enhances STEM teaching and learning but also 
helps students develop cognitive capabilities by practicing engineering design as 
a problem-solving strategy (Lammi & Becker, 2013). 

One measure used to investigate students’ cognitive approaches is the think- 
aloud protocol. Atman and Bursic (1998) employed the think-aloud protocol 
method as an evaluation tool to assess students’ design and problem-solving 
capacity. They used it to understand how undergraduate engineering students 
solved open-ended, ill-defmed engineering-design problems. Similar to Atman 
and Bursic’s studies, this study used a concurrent think-aloud (CTA) protocol in 
an elementary setting to inform technology education and STEM education 
about how elementary students solve design problems. 

As a part of Science Learning through Engineering Design (SLED), a Math 
Science Targeted Partnership (MSP) funded by the National Science Foundation 
(NSF), we conducted two studies in which we collected data from CTA sessions 
to measure students’ problem-solving ability. In the first study, data were 
collected on Cohort 1 in the 2011-2012 school year. In the second study, which 
is the subject of this article, data were collected on Cohort 2 in the 2012-2013 
school year. 
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In the original study of Cohort 1, we collected data from 33 CTA sessions 
to measure the students’ problem-solving ability in the 2011-2012 school year. 
Key features of engineering-design thinking often require many cognitive 
strategies; however, in the findings from Cohort 1, students showed limited use 
or no use of these strategies. The Cohort 1 findings revealed that the students 
spent very little time in computing (4%), managing (1%), testing (3%), and 
predicting results (4%). Students spent almost half of their time generating ideas 
(47%). CTA sessions from Cohort 1 indicate that student teams (triads) did not 
emphasize the use of computing (CO) and testing (TE) during the protocol 
sessions. Additionally, the cognitive strategy interpreting data (ID) was missing 
from all the protocol sessions. Even though mathematical reasoning skills such 
as computing, testing, and interpreting data are the key elements of engineering 
design, the results indicate that students were not using these skills. The results 
of the Cohort 1 study are compared with those of Cohort 2 in the Results 
section. 


Purpose of the Study 

The purpose of this study was to investigate how triads of students 
collaboratively developed solutions and applied scientific and mathematical 
concepts to inform their solution to engineering-design challenges. The 
questions guiding this study included the following: 

1. How do Grade 5 students conceptualize and learn design? 

2. Which aspects of the engineering-design process do students tend to 
emphasize? 

3. Which aspects of the engineering-design process do students tend to 
overlook? 

4. To what extend do students apply scientific concepts and mathematical 
reasoning when engaging in an engineering-design transfer problem? 

Theoretical Perspective 

The theoretical perspective for studying participants’ cognitive strategies 
through design is based upon the construct of transfer of learning (Bransford, 
Brown, & Cocking, 1999). Transfer of learning suggests that students can 
transfer their prior knowledge, skills, and experiences to new situations. When 
students are presented with new opportunities that are similar to pre-existing 
experiences, learning transfer can occur. Learning transfer is an indicator of 
understanding. Royer (1986) further describes the concept of transfer of 
learning: “Used as an index of understanding is equivalent to the idea that the 
ability to transfer learned information is evidence that understanding is present” 
(p. 95). In this study, we carefully crafted transfer problems that were similar in 
structure and scope to those presented to the students during a prior learning 
experience in order to assess a near transfer of learning (Thorndike, & 
Woodworth, 1901; Bransford, et al., 1999). We observed and coded student 
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dialogue to determine if students transferred what they had learned during the 
SLED activities to the transfer problems. Specifically, we were looking for 
students to transfer key engineering and science practices, scientific concepts, 
and the use of mathematical reasoning. 

Literature Review 

A CTA protocol is a procedure that allows a researcher to study the verbal 
report of one individual or group of individuals speaking their thoughts while 
engaging in an assigned task or problem. Recently, the CTA method has been 
applied to a wide variety of contexts, such as studying human operations of 
process controlling systems (Sanderson, Verhage, & Fuld, 1989), cognitive 
studies on writing (Ransdell, 1995) and reading (Pressley & Afflerbach, 1995). 
CTA protocols are endorsed as a promising tool to capture cognitive and 
metacognitive thinking in engineering education research (Atman & Bursic, 
1998). Multiple CTA studies have investigated engineering-design approaches 
within engineering education (Atman, Chimka, Bursic, & Nachtmann, 1999; 
Atman, Cardella, Turns, & Adams, 2005; Gainsburg, 2015) and team-based 
engineering design and problem solving (Mentzer, 2014; Stempfle & Badke- 
Schaub, 2002). 

However, investigating the cognition of designers during design is 
challenging. Ericsson and Simon (1993) suggest that CTA methods may provide 
the most authentic approach to achieve a record of cognitive activity during 
design because the designer is allowed to perform in his or her natural state of 
mind not altered by outside influences beyond verbalizing thoughts. Unlike 
structured elicitation approaches to cognitive investigations, CTA investigation 
seeks to place the participant in his or her most natural state of design thinking 
during the protocol sessions (van Someren, Barnard, & Sandberg, 1994). 

Some questions have arisen regarding CTA as a proper method to capture 
all aspects of design cognition. Lloyd, Lawson, and Scott (1995) reported that 
CTA methods may accurately capture short-term thought processes but fail to 
capture long-term states of memory. However, allowing designers to express 
ideas graphically allows for both short-term and long-term cognition (Ullman, 
Wood, & Craig, 1990). In addition, the CTA method requires participants to use 
their own language and to approach the assigned task as they would naturally 
solve it. Furthermore, some researchers questioned the validity of CTA data 
from young children. However, van Someren, Barnard, and Sandberg (1994) 
found that 

In our experience, the quality of verbalizations is not strongly associated 
with other properties that can easily be observed or measured. One possible 
exception is age. Young children usually find it difficult to think aloud. It is 
not clear if this is due to their verbalization skills, to the content of their 
thought processes or to the general difficulty of concentrating on a problem- 
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solving task. (p. 35-36) 

In a usability study, Donker and Markopoulos (2002) stated: “We expected 
methods like think-aloud that require high verbalization skills to be less 
effective for younger children or children with fever verbalization skills. Our 
expectations were not confirmed” (p. 314). We acknowledge these possible 
limitations of CTA protocols and, therefore, provided participants with the 
opportunity to create design sketches during the protocol sessions and to allow 
participants to work collaboratively and in their most natural state. 

Research Design 

Context of the Study 

This study is part of an NSF-funded MSP entitled SLED (for more 
information, see https://stemedhub.org/groups/sled). The collaborative 
partnership is made up of four colleges within a large, research-intensive 
university and four school corporations located in the north-central Midwest. 
The primary goal of the SLED project is to improve achievement in Grades 3-6 
students’ science learning through an engineering-design pedagogical approach. 
Over the course of 5 years, approximately 100 preservice teachers, 200 in- 
service teachers, and 5,000 students in Grades 3-6 will participate in the 
partnership. 

This research study was drawn from two SLED partnership schools. School 
Site 1 was located in an emerging urban school district, and School Site 2 was 
located in a rural fringe school district (see Table 1 for demographics). 

Table 1 


Demographics of School Sites 1 and 2 


Category 

School Sitel 

School Site 2 

Enrollment 

552 

124 

Ethnicity 

White/Caucasian 

56% 

76.6% 

Hispanic 

27.7% 

12.1% 

Black/non-Hispanic 

10.1% 

4.8% 

Asian 

0.4% 

0.0% 

Multiracial 

5.1% 

5.6% 

American Indian 

0.5% 

0.8% 

Free or reduced-price lunch 

71.9% 

43.6% 
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Student Design Activities 

In the second year of the project, the design team developed two math- 
embedded SLED activities to provide students’ mathematical reasoning practice. 
These activities were also designed to address science standards. The activities 
were (a) the CO 2 Device activity in which student designed a device utilizing a 
balloon filled with carbon dioxide and (b) the Recycling Paper activity. Table 2 
gives an overview of the design activities (see Appendices A and B for the 
design activity prompts). The series of science lessons implemented to support 
the engineering-design tasks was between five to seven 45-minute lessons 
delivered by SLED teachers. These science inquiry lessons contained the 
science content knowledge required to successfully complete the engineering- 
design tasks. These lessons targeted students’ misconceptions regarding 
conservation of mass, which have been documented by Driver (1983) and 
Driver, Squires, Rushworth, and Wood-Robinson (1994). 

Table 2 

Overview of the Two New SLED Design Activities for the Grade 5 Conservation 
of Mass Focused Design Tasks 

Title CO 2 Device Recycling Paper 


Description The CO 2 device activity 

required students to design a 
device utilizing a balloon 
filled with carbon dioxide 
generated from mixing 
vinegar and baking soda. 


The recycling paper activity 
involved students calculating 
the volume and mass of an 
irregular material (pile of 
shredded paper) or mixture of 
paper and water (sludge) while 
designing a process to make 
recycled paper. 


Participants 

During the 2012-2013 school year (Cohort 2), we collected data from a 
total of 66 CTA sessions. Analysis of data from the 66 sessions provided general 
design patterns of the cognitive approach that students took in the engineering- 
design challenges. Data from six sessions were further analyzed to understand 
how students used cognitive strategies to solve math-embedded design 
problems. 

We used criterion sampling to select cases that satisfied a specific criterion 
(Gall, Gall, & Borg, 2007). Participants for the think-aloud protocols were 
purposefully selected by the SLED teachers. Teacher recommendations were 
based upon (a) the students’ ability to express themselves verbally, (b) their 
ability to successfully function as contributing members of a design team, (c) 
their assent to participate in the study, and (d) parental consent for the students 
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to participate. Triads of student design teams were formed for each SLED 
classroom participating in the research. Welch (1999) suggests that pairing or 
grouping student participants allows for the design process to emerge naturally 
because most design efforts occur in groups of two or more people working 
together. Table 3 lists the total classroom size and genders of the three students 
selected as part of the triad for the six case studies. For example, Classroom 1 
had a total of 54 students, and one male student and two female students were 
selected to form a triad. 

Table 3 

Think-Aloud Participants: Classroom Size and Student Gender Demographics of 
Triads 

School Site 1 School Site 2 


School Classroom Classroom Classroom Classroom Classroom Classroom 
1 2 3 4 5 6 


Classroom 54 55 48 59 30 29 

size 

Student 1 M, 2 F 2M, IF 2M, IF 1M, 2F 2M, IF 2M, IF 
gender 


Data Collection 

Concurrent think-aloud protocol. The study employed the CTA protocol 
to capture students’ cognitive thinking processes and thoughts. After each 
participant classroom completed the SLED design activity, we selected a triad of 
students to participate in the CTA protocol. According to the Ericson and 
Simon’s (1993, p. 18) suggestion for CTA data collection, we provided students 
with two guidelines: (a) explain their thoughts directing their attention to the 
problem-solving procedure and (b) utilize their prior knowledge from the 
classroom-based design activity to the transfer problem presented in the protocol 
session. 

Transfer problem. The transfer problem was a key instrument used to 
provide each triad with the opportunity to study design problems similar to the 
SLED design activities. As Cross (1994) suggested, transfer problems consist of 
three parts: (1) a goal, (2) constraints to address, and (3) design criteria to gauge 
the final design solution against. In this study, we focused on creating authentic 
engineering-design problems that required the use of science concepts 
embedded within the task. The problem scenarios were created based on 
situations that students might encounter in their daily lives or on designing 
products that were familiar to them. One transfer problem, Scotty’s Scooters in 
Appendix C, was created for both the Recycling Paper and the CO 2 Device tasks 
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because they addressed the same science concept, conservation of mass. 

SLED knowledge assessments. To investigate the gap between knowing 
and applying scientific knowledge, we adopted a set of pre- and post-knowledge 
tests. Using an approach similar to Singer, Marx, Krajcik, and Chambers (2000) 
and Fortus, Dershimer, Krajcik, Marx, and Mamlok-Naaman (2004), we 
constructed a multiple-choice test that contained items of low, medium, and high 
cognitive demand to assess students’ preexisting knowledge and to measure 
gains in knowledge. As Fortus et al. (2004) described, in order to get accurate 
indication of student’s growth in knowledge from the SLED activities, 
researchers must first determine what students already know about the science. 
Pretest assessments were administered at the start of the academic year, and 
posttests were administered within 10 days of completing the SLED activity. 
Because one of the participants was absent when the pretest assessment was 
administered, pre- and post-test scores were only available for 17 participants. 

Data Analysis 

Think-aloud protocol analysis. The study adopted Halfm’s (1973) codes 
to analyze the think-aloud data. These codes were created during Halfm’s 
Delphi study that researched commonly used cognitive strategies by successful 
professional scientists, engineers, and inventors. Seventeen cognitive strategies 
were generated, and detailed descriptions were developed from the research and 
validated by 28 panel members. One advantage to using Halfm’s codes for this 
analysis is that it provides problem-solving processes as well as comprehensive 
cognitive strategies usually used in design activities. Halfm’s coding scheme 
allowed us to investigate students’ abilities to apply their design and problem¬ 
solving capabilities to transfer problems (Hill, 1997; Kelley, 2008; Kelley, 
Capobianco, & Kaluf, 2015; Wicklein, 1996). 

Interrater reliability of think-aloud analysis. Several steps were followed 
to ensure interrater reliability when analyzing the video data. First, two 
researchers carefully reviewed the coding scheme and definitions created by 
Halfin (1973, pp. 135-204) and mapped students’ dialogues to these codes. 
Seven codes were determined to be outside the parameters of the protocol 
sessions, so these codes were removed from the coding list for the purpose of 
this research, i As a result, we utilized 10 of the 17 codes developed by Halfin 
(1973). The 10 codes used in this study are described in Appendix D. Second, 
sample video clips were viewed by the two researchers together, and their 
interpretations of the selected segments were discussed in order to reach coding 
consensus. Finally, the two researchers independently coded the video segments, 
and their analyses were compared. The kappa coefficient for interrater reliability 
was calculated. Hruschka et al. (2004) suggested that at least 20% of the data set 


i Due to the time limitations for the protocol sessions, participants were not able 
to construct models or conduct experiments. 
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results should be compared between two independent researchers. In this study, 
we analyzed one third of the CTA sessions to ensure interrater reliability. Each 
video from the six CTA sessions was segmented into three parts, and one 
segment from each session was analyzed for interrater reliability. To achieve 
acceptable levels of intercoder reliability, we followed Hruschka et al.’s iterative 
coding method (p. 311), and a Kappa coefficient of 0.91 was calculated using 
NVivo (Version 10) with 99.45% agreement. 

SLED knowledge assessments analysis. In order to measure that 
participants successfully gained scientific and mathematical knowledge through 
the SLED engineering-design lessons, we compared pre- and post-test scores 
using a paired sample /-test using SAS (Version 9.4), a statistical analysis 
software. 


Results 

Concurrent Think-Aloud Protocols 

Data from 66 concurrent think-aloud protocols were collected during the 
2012-2013 school year (Cohort 2). The mean percentages for the coded sessions 
are displayed in Figure 1. 

QH: MA: TE: ID: 



Figure 1. Result of total group mean percentages per code in Cohort 2. 
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Based upon the coded data for Cohort 2, we found that: 

a) Participants continued to effectively define the problem and identify 
constraints and criteria when compared to the results from Cohort 1. 

b) Students spent, on average, 50.4% of their time designing (DE), 14.4% 
of the time analyzing (AN), and 32.4% of the time defining the problem 
(DF). These percentages were comparable to the Cohort 1 findings. 

c) Eleven sessions involved student dialogues that included numerical data, 
estimating, and mathematical predictions compared to zero sessions 
from Cohort 1. One reason is that design activities such as the CO 2 
Device and Recycling Paper activities contain numerical data, 
estimations, volume, or surface area within the design brief. We also 
noticed that the overall length of several protocol sessions that included 
computing dialogue. In some cases, this was due to students focusing on 
computing numbers instead of designing solutions. 

We further investigated how students engaged in math-embedded design 
tasks to determine if learning transfer occurred accurately and if students 
demonstrated proficiency in the key science standards. Six CTA sessions were 
selected from the CO 2 Device and Recycling Paper activities to further 
investigate the dialogue of triads within the time they spent computing. For 
School Site 1, CTA sessions from the CO 2 Device activity were chosen for all 
three classrooms (Classrooms 1, 2, and 3). For School Site 2, a CTA session 
form the Recycling Paper activity was chosen for one classroom (Classroom 4), 
and CTA sessions from the CO 2 Device activity were chosen for the other two 
classrooms (Classrooms 5 and 6). Figure 2 shows the coded analysis of each 
CTA protocol session as a percentage of time; the segment representing 
computational thinking is labeled CO (dark shading with dots). The six sessions 
selected showed that students spent 11% to 46% of their time on computational 
thinking (Figures 2). 
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School Site 1 


Classroom 1 

CO 2 Device Activity 
MA ID AN DE 

Z 3% 6% 5 o /o 28 



Classroom 2 

CO 2 Device Activity 
pR QH MA | D AN 



MO DF 
1 % 14 % 


Classroom 3 

CO 2 Device Activity 

OH MA 
PR , 1% 





School Site 2 

Classroom 4 

Classroom 5 

Classroom 6 


Recycling Paper 
Activity 
ID AN 

QH 9% 1% 

4% 



co 
26% 

MO DF 
2 % 8% 


CO 2 Device Activity 

Q H MA an DE 
5% 2% 14% 



CO 2 Device Activity 



Figure 2 . Percent of time spent on each cognitive strategy from School Sites 1 
and 2. 
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Science and engineering-design knowledge test. Seventeen participants 
took pre- and post-test exams to measure knowledge gained by participating in 
the engineering-design activity. A paired-sample t-test was performed to 
determine if these gains were statistically significant. The mean scores for the 
pretest and posttest and the results of the paired t-test are shown in Table 4. The 
paired t-test indicates that the sample means from pre- to post-test are 
significantly different at the p < 0.0002 level. 


Table 4 

Paired t-test Result from Knowledge Test Scores 



Pretest 

Posttest 

95% CL for 



n 

M 

SB 

M 

SD 

mean 

difference 

DF 

t p 

17 

9 

1.90 

11.94 

1.72 

1.85,3.78 

16 

4.88 < 

0.0002 


Applying science and math concepts to engineering-design task. The 

primary science concept being taught during the Recycling Paper and CO 2 
Device design tasks was conservation of mass. In order to assess students’ 
ability to apply the science concept to engineering-design tasks, we included a 
prompt in the transfer problem asking “What is the mass of the re-designed 
scooter?” We analyzed the computing (CO) segments of the CTA sessions for 
the triad’s discussion regarding conservation of mass. Two of the six triads 
(Classrooms 1 and 2) correctly answered that the mass is the same, three triads 
(Classrooms 3, 4, and 5) tried to determine a new mass, and one triad 
(Classroom 6) did not address the question regarding conservation of mass. In 
order to illustrate this, we include the dialogue and sketches for two of the triads 
here: one triad who answered the question correctly, and one who did not. The 
dialogue from the triad from Classroom 1, who correctly answered the question 
regarding conservation of mass, appears below, and their sketch for their 
solution is shown in Figure 3. 


Student A: 

What is the mass? 

Student B: 

The scooter mass is ... 

Student C: 

Wait... the mass does not change. 

Student A: 

Yes, mass does not change. It is gonna [sic] be the 
same the scooter mass. 

Student B: 

[what about] not the tires? 

Student A: 

No, the tires are the part of the scooter. It won’t 
change 
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Figure 3. Sketch from the Classroom 1 triad. 


Although the triads in Classrooms 1 and 2 answered the question 
correctly, three triads forgot that the mass does not change even when 
the scooter is collapsed or disassembled. The triad in Classroom 5 drew 
the sketch shown in Figure 4, which illustrates their calculations, and 
their dialogue from that session appears below. 


Student X: 


Student Y 
Student X 
Student Y 
Student X 


We need to figure out what the mass could be. ... Old one was 
70 cm, 80 cm, and 15 cm. So, we do, 55 times 40 and 15... 
[Calculating numbers, they multiplied height by width and 
depth]. The scooter ... Yes I got 30,000 pound? 

Pound? It is mass. 

It would be gram? 

No, kilogram? 30,000 kg. 

That’s heavy, [sigh] 


- 95 - 












Journal of Technology Education Vol. 28 No. 2, Spring 2017 



Figure 4. Sketch from the Classroom 5 triad. 


The dialogue from the students Classroom 5 showed three types of 
mathematical and science errors. First, the students failed to recall the 
concept that the mass does not change even though the physical 
appearance of the object has changed. Second, to determine the mass of 
the redesigned scooter, they drew a box around the scooter and 
multiplied the dimensions of the box. This indicates that they did not 
understand the difference between the concept of mass and the concept 
of volume. Third, the value calculated for the mass was incorrect (it 
contained mathematical errors). 


Discussion 

The results from the math-embedded tasks demonstrated that students spent 
additional time engaged in computational thinking during the protocol sessions. 
However, it also revealed that some students still struggled to accurately transfer 
science concepts especially conservation of mass. One triad from School Site 1 
and two triads at School Site 2 (Classrooms 4, 5, and 6) attempted to calculate a 
new mass for the disassembled scooter, which showed that they did not 
recognize that mass is conserved. Two of the three triads at School Site 1 
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(Classrooms 1 and 2) did correctly identify that mass was conserved when the 
scooter was disassembled. These results differ from the knowledge test results 
which indicated gains for all students from pre- to post-test. The results indicate 
that students were successful at identifying the concept of conservation of mass 
on a multiple-choice test but most of the students were unable to transfer it to a 
new situation. The results should be used by stakeholders within STEM 
education who seek to improve learning through engineering to help students 
use numerical data to inform their design decisions. Furthermore, using CTA 
protocols as a form of assessment for design thinking and problem solving 
revealed gaps in understanding that were not evident from the pre-and post-test 
knowledge assessments. We believe that using CTA protocols effectively 
assesses students’ abilities to apply or transfer this knowledge to different 
situations. 


Limitations 

We acknowledge that there are some limitations to this research. First, we 
acknowledge that the criterion sampling approach may not provide sampling 
that best represents the average ability of the student body from each classroom 
due to potential bias of the teacher when selecting participants for this study. 
Second, the concurrent think-aloud methodology is a qualitative approach to 
study individual cases; therefore, these findings cannot be generalized to the 
entire population. We acknowledge that additional attention should be given to 
using alternative data methods such as open-response questions within 
knowledge tests in addition to the think-aloud protocol in order to strengthen the 
assessment of students’ knowledge of science content. 

Conclusions and Implications 

The purpose of this study was use a CTA protocol methodology in order to 
classify participants’ cognitive approaches to problem solving while engaged in 
an engineering-design task. Verbal data collected from each CTA session were 
categorized and organized using Halfm’s (1973) codes to help us identify 
strategies of problem-solving and design-thinking skills. Additionally, we 
sought to locate within the protocol places where the transfer of learning of 
science concepts were present and to assess the accuracy of this transfer. The 
findings from this study revealed that participants were able to apply numerous 
cognitive strategies while creating design solutions and working in triads. This 
research confirmed results from previous studies finding that students were able 
to navigate through the design process moving from the problem space to 
solution space (Kruger & Cross, 2006) and not getting “stuck” in either space 
(Kelley, Capobianco, & Kaluf, 2015). Results from the six case studies revealed 
that students increased the time spent on computational thinking when given 
math-embedded design tasks. 
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Although SLED teachers used techniques such as Predict, Observe, Explain 
(POE) investigations suggested by educational researchers to help overcome 
misconceptions regarding the law of conservation of mass (Dial, Riddley, 
Williams, & Sampson, 2009), misconceptions remained for some students. It is 
not enough for students to know basic science, math, and engineering practices; 
it is important for students to know how to apply their knowledge and skills to 
solve real-world problems. We believe that the findings from this study provide 
strong rationale to use CTA protocol methods to assess student’s abilities to 
apply their knowledge, skills, and practice to transfer problems set in scenarios 
with real-world contexts. 

Elementary science teachers using engineering design as an approach to 
improve science learning should provide additional opportunities for students to 
improve their ability to transfer science and mathematical reasoning beyond the 
initial design tasks. Some suggest that mathematical reasoning needs to move 
beyond traditional classroom practices (Lesh & Yoon, 2004), requiring students 
to consider different approaches to thinking when problems are posed or 
requiring them to transfer their learning to real-world problems (Chamberlin & 
Moon, 2005). Elementary teachers could also use Model Eliciting Activities 
(Diefes-Dux, Moore, Zawojewski, Imbrie, & Follman (2004) to help students’ 
practice applying mathematical thinking and spatial reasoning to solve problems 
in a similar way to the transfer problem used in this research study. Additional 
research needs to be conducted to better understand the results from this study. 
The findings from this study help to shed new light on (a) the complexities of 
knowledge transfer, (b) the limits of students’ mathematical reasoning, and (c) 
how the engineering-design approach to teaching science provides new 
challenges and new opportunities to promote STEM education. 

Secondary technology educators must be prepared for students to enter their 
classrooms with preexisting knowledge of and experience with the engineering- 
design process due to the Next Generation Science Standards. Additional 
technology educators should seize the opportunity to put students in new and 
novel situations that require them to use their math and science knowledge while 
engaging in engineering design. We hope that technology educators will use 
these research findings to adapt and refine their own engineering-design 
curriculum. Technology educators can leverage the new opportunities of 
engineering design within science education to reinforce the application of 
science and mathematical thinking in technology education classrooms; this is 
one way to continue to position technology education within STEM education. 


- 98 - 



Journal of Technology Education 


Vol. 28 No. 2, Spring 2017 


Acknowledgements 

This article is based upon work supported by the National Science 
Foundation under Grant No. (DUE 0962840). Any opinions, findings, and 
conclusions or recommendations expressed in this material are those of the 
authors and do not necessarily reflect the views of the National Science 
Foundation. 


References 

Atman, C. J., & Bursic, K.M. (1998). Verbal protocol analysis as a method to 
document engineering student design processes. Journal of Engineering 
Education, 87(2), 121-132. doi:10.1002/j.2168-9830.1998.tb00332.x 
Atman, C. J., Chimka, J. R., Bursic, K. M., & Nachtmann, H. L. (1999). A 
comparison of freshman and senior engineering design processes. Design 
Studies, 20(2), 131-152. doi:10.1016/S0142-694X(98)00031-3 
Atman, C. J., Cardella, M. E., Turns, J., & Adams, R. (2005). Comparing 

freshman and senior engineering design processes: An in-depth follow-up 
study. Design Studies, 26(4), 325-357. doi:10.1016/j.destud.2004.09.005 
Bransford, J. D., Brown, A. L., & Cocking, R. R. (Eds.). (1999). How people 
learn: Brain, mind, experience, and school Washington, DC: National 
Academy Press. doi:l0.17226/9853 

Chamberlin, S. A., & Moon, S. M. (2005). Model-eliciting activities as a tool to 
develop and identify creatively gifted mathematicians. The Journal of 
Secondary Gifted Education, 77(1), 37-47. doi: 10.4219/jsge-2005-393 
Cross, N. (1994). Engineering design methods: Strategies for product design 
(2nd ed.). Chichester, England: Wiley & Sons. 

Dial, K., Riddley, D., Williams, K., & Sampson, V. (2009). Addressing 

misconceptions: A demonstration to help students understand the law of 
conservation of mass. The Science Teacher, 76(1), 54-57. 

Donker, A., & Markopoulos, P. (2002). A comparison of think-aloud, 
questionnaires and interviews for testing usability with children. In X. 
Faulkner, J. Finlay, & F. Detienne (Eds.), People and computers XVI— 
Memorable yet invisible: Proceedings ofHCI 2002 (pp. 305-316). London, 
United Kingdom: Springer-Verlag. doi: 10.1007/978-1 -4471-0105-5 18 
Diefes-Dux, H. A., Moore, T., Zawojewski, J., Imbrie, P. K., & Follman, D. 
(2004, October). A framework for posing open-ended engineering 
problems: Model-eliciting activities. In Frontiers in Education, 2004. FIE 
2004. 34th Annual (pp. F1A-3). IEEE. 

Driver, R. (1983). The pupil as scientist? Milton Keynes, England: Open 
University Press. 

Driver, R., Squires, A., Rushworth, P., & Wood-Robinson, V. (1994). Making 
sense of secondary science: Research into children’s ideas. Abingdon, 
England: Routledge. 


- 99 - 





Journal of Technology Education Vol. 28 No. 2, Spring 2017 


Ericsson, K. A., & Simon, H. A. (1993). Protocol analysis: Verbal reports as 
data. Cambridge, MA: MIT Press. 

Fortus, D., Dershimer, R. C., Krajcik, J., Marx, R. W., & Mamlok-Naaman, R. 
(2004). Design-based science and student learning. Journal of Research in 
Science Teaching , 47(10), 1081-1110. doi:10.1002/tea.20040 

Gainsburg, J. (2015). Engineering students' epistemological views on 

mathematical methods in engineering. Journal of Engineering Education, 
104(2), 139-166. doi: 10.1002/jee.20073 

Gall, M. D., Gall, J. P., & Borg, W. R. (2007). Educational research: An 
introduction (8th ed.). Boston, MA: Pearson Education. 

Halfm, H. H. (1973). Technology: A process approach. (Doctoral dissertation). 
Available from ProQuest Dissertations and Theses database. (UMI No. 
7323867) 

Hill, R. B. (1997). The design of an instrument to assess problem solving 

activities in technology education. Journal of Technology Education, 9(1), 
31-46. doi:10.21061/jte.v9il.a.3 

Hill, R. B. (2006). New perspectives: Technology teacher education and 

engineering design. Journal of Industrial Teacher Education, 43(3), 45-63. 
Retrieved from http://scholar.lib.vt.edu/ejournals/JITE/v43n3/pdf/hill.pdf 

Hruschka, D. J., Schwartz, D., St. John, D. C., Picone-Decaro, E., Jenkins, R. 

A., & Carey, J. W. (2004). Reliability in coding open-ended data: Lessons 
learned from HIV behavioral research. Field Methods, 16(3), 307-331. 
doi: 10.1177/1525822X04266540 

Kelley, T. R. (2008). Cognitive processes of students participating in 

engineering-focused design instruction. Journal of Technology Education, 
19(2), 50-64. 

Kelley, T. R., Capobianco, B. M., & Kaluf, K. J. (2015). Concurrent think-aloud 
protocols to assess elementary design students. International Journal of 
Technology and Design Education, 25(4), 521-540. doi: 10.1007/s 10798- 
014-9291-y 

Kruger, C., & Cross, N. (2006). Solution driven versus problem driven design: 
Strategies and outcomes. Design Studies, 27(5), 527-548. 
doi: 10.1016/j .destud.2006.01.001 

Lammi, M., & Becker, K. (2013). Engineering design thinking. Journal of 
Technology Education, 24(2), 55-77. doi: 10.2106l/jte.v24i2.a.5 

Lesh, R., & Yoon, C. (2004). Evolving communities of mind—In which 

development involves several interacting and simultaneously developing 
strands. Mathematical Thinking and Learning, 6(2), 205-226. 
doi:10.1207/sl5327833mtl0602_7 

Lewis, T. (2005). Coming to terms with engineering design as content. Journal 
of Technology Education, 16(2), 37-54. doi: 10.2106 l/jte.vl6i2.a.3 


- 100 - 



Journal of Technology Education 


Vol. 28 No. 2, Spring 2017 


Lloyd, P., Lawson, B., & Scott, P. (1995). Can concurrent verbalization reveal 
design cognition? Design Studies, 16(2), 237-259. doi: 10.1016/0142- 
694X(94)00011-2 

Mentzer, N. (2014). Team based engineering design thinking. Journal of 
Technology Education. 25(2), 52-72. doi:10.21061/jte.v25i2.a.4 

National Research Council. (2012). A framework for K-12 science education: 
Practices, crosscutting concepts, and core ideas. Washington, DC: National 
Academies Press, doi: 10.17226/13165 

NVivo qualitative data analysis Software (Version 10) [Computer software]. 
(2012). Melbourne, Australia: QSR International. 

Pressley, M., & Afflerbach, P. (1995). Verbal protocols of reading: The nature 
of constructively responsive reading. Hillsdale, NJ: Erlbaum. 

Ransdell, S. (1995). Generating thinking-aloud protocols: Impact on the 
narrative writing of college students. American Journal of Psychology, 
108(1), 89-98. doi: 10.2307/1423102 

Robelen, E. W. (2103). Engineering building a foundation in K-12 curricula. 
Education Week, 32(26), 1, 18-19. 

Royer, J. M. (1986). Designing instruction to produce understanding: An 
approach based on cognitive theory. In G. D. Phye & T. Andre (Eds.), 
Cognitive classroom learning: Understanding, thinking, and problem 
solving (pp. 83-113). Orlando, FL: Academic Press. 

Sanderson, P. M., Verhage, A. G., & Fuld, R. B. (1989). State-space and verbal 
protocol methods for studying the human operator in process control. 
Ergonomics, 32(11), 1343-1372. doi:10.1080/00140138908966911 

SAS (Version 9.4) [Computer software]. (2012). Cary, NC: SAS Institute. 

Singer, J., Marx, R. W., Krajcik, J. S., & Chambers, J. C. (2000). Designing 
curriculum to meet national standards. Paper presented at the Annual 
Meeting of the American Educational Research Association, New Orleans, 
LA. Retrieved from 

http://www.umich.edu/~hiceweb/papers/2000/designing_curriculum_to_me 

et/singer_NARST2000_doc.pdf 

Stempfle, J., & Badke-Schaub, P. (2002). Thinking in design teams—An 
analysis of team communication. Design Studies, 23(5), 473-496. 
doi: 10.1016/SO 142-694X(02)00004-2. 

Thorndike, E. L. and Woodworth, R. S. (1901). The influence of improvement 
in one mental function upon the efficiency of other functions, 
Psychological Review 8: Part I, pp. 247-261. 

Ullman, D. G., Wood, S., & Craig, D. (1990). The importance of drawing in the 
mechanical design process. Computers & Graphics, 14(2), 263-274. 
doi: 10.1016/0097-8493(90)90037-X 

van Someren, M. W., Barnard, Y. F., & Sandberg, J. A. C. (1994). The think- 
aloud method: A practical guide to modelling cognitive processes. London, 
England: Academic Press. 


- 101 - 





Journal of Technology Education Vol. 28 No. 2, Spring 2017 


Welch, M. (1999). Analyzing the tacit strategies of novice designers. Research 
in Science & Technical Education , 77(1), 19-34. 
doi: 10.1080/0263 514990170102 

Wicklein, R. C. (1996). Processes of a technologist: Key curriculum component 
for technology education. Unpublished manuscript. 

Wicklein, R. C. (2006). Five good reason for engineering design as the focus for 
technology education. The Technology Teacher , 65(7), 25-29. 

About the Authors 

Todd Kelley (trkelley@purdue.edu) is Associate Professor in the 
Engineering/Technology Teacher Education Program in the Department of 
Technology Leadership and Innovation at Purdue University. 

Euisuk Sung (sunge@purdue.edu) is a PhD Candidate in the 

Engineering and Technology Teacher Education Program in the Department of 

Technology Leadership and Innovation at Purdue University. 


- 102 - 



Journal of Technology Education 


Vol. 28 No. 2, Spring 2017 


APPENDIX A 


Design Task 1: Recycling Paper 


SLED 

Science Learning through 
Engineering Design 


Recycling Paper for Your School 

The greater Lafayette area is facing the problem of increased paper waste. The 
city of Lafayette is interested in recycling the paper waste. They need your help 
to design a strainer system for the recycling plant that will produce very thin 
recycled paper. 

Criteria 

• Paper produced should be as thin as possible 

• Paper should have equal or consistent thickness throughout the paper 

• There should not be any holes on the paper 

• Paper should be at least 3”x 5” 

• Use 2.5 liters of water 

Constraints 

• You can only use the materials, tools, and paper available to you in the 
class 

• Paper blending has to be done only by your teacher 

Deliverables 

■ A dry recycled paper that has dimensions of 3 inches by 5 inches. 

*The design task was developed by §enay Purzer, Venkatesh Merwade, Brad 
Harriger, David Eichinger, and Erin Doherty. 
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APPENDIX B 
Design Task 2: CO 2 Device 



An employee of the Indiana Sand Dunes Chemical Company noticed that a 
byproduct (a substance made during a reaction, but not used) of the chemical 
process he was developing was a gas. In fact, the employee noted that a lot of 
this gas was formed after combining two reactants, vinegar and baking soda. 
The gas, carbon dioxide, was enough to inflate a balloon. The Indiana Sand 
Dunes Chemical Company is convinced that the production of the gas can be 
used to make a useful product and they are asking you to help them design a 
product that people would want to buy and use. Your team is limited to one 
balloon filled with gas. 

You may use the following materials to generate the amount of gas necessary 
for your device: 

• One 16 - 24oz empty plastic drink bottle (avoid wide-necks) 

• 12” helium quality balloon 

• 200mL distilled white vinegar 

• 3 level teaspoons (or 1 level tablespoon) Baking Soda (15 - 20g) 

• Bag clip 


* The design task was developed by Kari Clase, Melissa Colonis, John Grutzner, 
Bryan Hubbard, Alyssa Panitch, and Nancy Tyrie. 


- 104 - 







Journal of Technology Education 


Vol. 28 No. 2, Spring 2017 


APPENDIX C 

SLED 

Science Learning through 

Transfer Problem: Scotty’s Scooters Engineering Design 




The Problem 

The owner, Scotty, has a brand new line of scooters that he has designed and 
must package for shipment. However, storage at Scotty’s Scooters is limited. 
Scotty realizes that fully assembled packaged scooters take up too much 
space. A re-design of the scooters is necessary to allow them to collapse or 
break apart to fit in smaller boxes. Scotty has asked for your design team to 
help in re-designing his scooter so that it will collapse and fit in the smallest 
packaging as possible. 

Scotty is looking for the following re-design features: 

• The scooter design must collapse or break apart. 

• All pieces MUST be in the shipping package. 

• The package must take up as little storage space as possible. 

Scooter Facts: 

• The size of the fully assembled scooter has a length of 70 cm, a 
height of 80 cm, and a depth of 15 cm. 

• The scooter’s mass is 3.5 kg. 

• The fully assembled scooter fits in a box 75 cm long x 90 cm high x 
20 cm deep. 

Scotty’s questions about your re-design: 

• What is the size of the box to hold the re-designed (collapsed) 
scooter? 

• What is the mass of the re-designed scooter? 
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• How much space can you save with the re-designed scooter? 

Your Task 

Describe how you would re-design a scooter that will collapse or break apart 
to create smaller shipping packages. 

• Please describe aloud how you would start the design task - where 
would you begin? 

• How would you design to include all the features listed above? 

• How would you answer Scotty’s questions? 
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APPENDIX D 


Cognitive Processes Identified by Halfin’s (1973) Study of High-level 

Designers 

(10 of the 17 total codes that emerged in the CTA sessions) 


Proposed 
mental methods 


Definition 

Analyzing 

AN 

The process of identifying, isolating, 
taking apart, breaking down, or 
performing similar actions for the 
purpose of setting forth or clarifying 
the basic components of a 
phenomenon, problem, opportunity, 
object, system, or point of view. 

Computing 

CO 

The process of selecting and applying 
mathematical symbols, operations, 
and processes to describe, estimate, 
calculate, quantity, relate, and/or 
evaluate in the real or abstract 
numerical sense. 

Defining problem(s) 

DF 

The process of stating or defining a 
problem which will enhance 
investigation leading to an optimal 
solution. It is transforming one state of 
affairs to another desired state. 

Designing 

DE 

The process of conceiving, creating 
inventing, contriving, sketching, or 
planning by which some practical 
ends may be effected, or proposing a 
goal to meet the societal needs, 
desires, problems, or opportunities to 
do things better. Design is a cyclic or 
iterative process of continuous 
refinement or improvement. 

Interpreting data 

ID 

The process of clarifying, evaluating, 
explaining, and translating to provide 
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(or communicate) the meaning of 
particular data. 

Managing 

MA 

The process of planning, organizing, 
directing, coordinating, and 
controlling the inputs and outputs of 
the system. 

Modeling 

MO 

The process of producing or reducing 
an act, or condition to a generalized 
construct which may be presented 
graphically in the form of a sketch, 
diagram, or equation; presented 
physically in the form of a scale 
model or prototype; or described in 
the form of a written generalization. 

Predicting 

PR 

The process of prophesying or 
foretelling something in advance, 
anticipating the future on the basis of 
special knowledge. 

Questions/hypotheses 

QH 

Questioning is the process of asking, 
interrogating, challenging, or seeking 
answers related to a phenomenon, 
problem, opportunity element, object, 
event, system, or point of view. 

Testing 

TE 

The process of determining the 
workability of a model, component, 
system, product, or point of view in a 
real or simulated environment to 
obtain information for clarifying or 
modifying design specifications. 
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